Information processor

ABSTRACT

An information processing system is provided. The information processing system includes first and second information processors. The first information processor obtains a first moving image including a face and detecting an eye from two or more first images among the first moving image. The first information processor detects a pupil from the eye detected from the first image, calculating its size, and performing learning using a change over time in the size of the pupil. The second information processor obtains a second moving image including a face and detecting an eye from two or more second images among the second moving image. The second information processor detects a pupil from the eye detected from the second image, calculating its size, and performing inference on the change over time in the size of the pupil on the basis of the result of the learning performed by the first information processor.

TECHNICAL FIELD

One embodiment of the present invention relates to an informationprocessor. Another embodiment of the present invention relates to aninformation processing system. Another embodiment of the presentinvention relates to an information processing method. Anotherembodiment of the present invention relates to an information terminal.

BACKGROUND ART

A user may feel fatigue, drowsiness, or the like when the user uses aninformation terminal such as a smartphone or a tablet for a long time.In particular, the user may feel eye fatigue by gazing at a screen ofthe information terminal for a long time. Patent Document 1 discloses adetection device and a detection method for eye fatigue.

A pupil diameter changes depending on whether there is fatigue,drowsiness, or the like. For example, when there is fatigue ordrowsiness, the pupil diameter becomes smaller than that in the casewhere there is no fatigue or drowsiness. The pupil diameter generallychanges periodically; however, in the case where there is fatigue ordrowsiness, the change cycle of the pupil diameter becomes longer thanthat in the case where there is no fatigue or drowsiness.

REFERENCE Patent Document

[Patent Document 1] Japanese Published Patent Application No.2017-169601

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

It is preferable that fatigue, drowsiness, or the like of the user beable to be detected in real time during the use of the informationterminal such as the smartphone or the tablet, in which case, forexample, the operation of the information terminal can be changedaccording to whether the user has fatigue, drowsiness or the like. Inthe case where fatigue, drowsiness, or the like of the user is detectedin real time, the information terminal in use itself preferably has afunction of presuming the fatigue, drowsiness, or the like of the user.However, in order to detect eye fatigue by the method disclosed inPatent Document 1, a dedicated device is necessary.

An object of one embodiment of the present invention is to provide aninformation processor having a function of detecting fatigue,drowsiness, or the like of the user in real time. Another object of oneembodiment of the present invention is to provide an informationprocessor having a function of presuming fatigue, drowsiness, or thelike of the user with high accuracy. Another object of one embodiment ofthe present invention is to provide an information processor having afunction of presuming fatigue, drowsiness, or the like of the user by asimple method. Another object of one embodiment of the present inventionis to provide an information processor having a function of presumingfatigue, drowsiness, or the like of the user in a short time.

An object of one embodiment of the present invention is to provide aninformation processing system having a function of detecting fatigue,drowsiness, or the like of the user in real time. Another object of oneembodiment of the present invention is to provide an informationprocessing system having a function of presuming fatigue, drowsiness, orthe like of the user with high accuracy. Another object of oneembodiment of the present invention is to provide an informationprocessing system having a function of presuming fatigue, drowsiness, orthe like of the user by a simple method. Another object of oneembodiment of the present invention is to provide an informationprocessing system having a function of presuming fatigue, drowsiness, orthe like of the user in a short time.

Note that the description of a plurality of objects does not precludethe existence of each object. One embodiment of the present inventiondoes not necessarily achieve all the objects described as examples.Furthermore, objects other than those listed are apparent fromdescription of this specification, and such objects can be objects ofone embodiment of the present invention.

Means for Solving the Problems

One embodiment of the present invention is an information processorincluding an imaging unit and an arithmetic unit having a functionperforming arithmetic operation by machine learning, in which theimaging unit has a function of obtaining a moving image that is a groupof images of two or more frames, the arithmetic unit has a function ofdetecting a first object from each of two or more of the images includedin the moving image, the arithmetic unit has a function of detecting asecond object from each of the detected first objects, the arithmeticunit has a function of calculating a size of each of the detected secondobjects, and the arithmetic unit has a function of performing machinelearning using a change over time in the size of the second object.

In the above embodiment, the machine learning may be performed with aneural network.

In the above embodiment, the moving image may include a face, the firstobject may be an eye, and the second object may be a pupil.

One embodiment of the present invention is an information processorhaving a function of performing inference on the basis of a learningresult obtained by performing learning using a change over time in asize of a first object shown in two or more first images included in afirst moving image. The information processor has a function ofobtaining a second moving image, the information processor has afunction of detecting a second object from each of two or more secondimages included in the second moving image, the information processorhas a function of detecting a third object from each of the detectedsecond objects, the information processor has a function of calculatinga size of each of the detected third objects, and information processorhas a function of performing inference on a change over time in the sizeof the third object on the basis of the learning result.

In the above embodiment, the learning and the inference may be performedwith a neural network, and the learning result may include a weightingcoefficient.

In the above embodiment, the first moving image may include a firstface, the second moving image may include a second face, the first andthird objects may be pupils, and the second object may be an eye.

In the above embodiment, the information processor may have a functionof presuming fatigue of a person including the second face.

Effect of the Invention

According to one embodiment of the present invention, an informationprocessor having a function of detecting fatigue, drowsiness, or thelike of the user in real time can be provided. According to anotherembodiment of the present invention, an information processor having afunction of presuming fatigue, drowsiness, or the like of the user withhigh accuracy can be provided. According to another embodiment of thepresent invention, an information processor having a function ofpresuming fatigue, drowsiness, or the like of the user by a simplemethod can be provided. According to another embodiment of the presentinvention, an information processor having a function of presumingfatigue, drowsiness, or the like of the user in a short time can beprovided.

According to one embodiment of the present invention, an informationprocessing system having a function of detecting fatigue, drowsiness, orthe like of the user in real time can be provided. According to anotherembodiment of the present invention, an information processing systemhaving a function of presuming fatigue, drowsiness, or the like of theuser with high accuracy can be provided. According to another embodimentof the present invention, an information processing system having afunction of presuming fatigue, drowsiness, or the like of the user by asimple method can be provided. According to another embodiment of thepresent invention, an information processing system having a function ofpresuming fatigue, drowsiness, or the like of the user in a short timecan be provided.

Note that description of the plurality of effects does not preclude theexistence of other effects. One embodiment of the present invention doesnot necessarily achieve all the effects described as examples. In oneembodiment of the present invention, other objects, effects, and novelfeatures will be apparent from the description of the specification andthe drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a structure example of aninformation processing system.

FIG. 2 is a flow chart showing an example of a method for operating aninformation processor.

FIG. 3 is a flow chart showing an example of a method for operating aninformation processor.

FIG. 4 is a flow chart showing an example of a method for operating theinformation processor.

FIG. 5A, FIG. 5B, and FIG. 5C are schematic views illustrating anexample of a method for operating the information processor.

FIG. 6A and FIG. 6B are schematic views illustrating examples of amethod for operating the information processor.

FIG. 7A1, FIG. 7A2, FIG. 7B1, and FIG. 7B2 are schematic viewsillustrating examples of a method for operating the informationprocessor.

FIG. 8A and FIG. 8B are schematic views illustrating an example of amethod for operating the information processor.

FIG. 9A and FIG. 9B are schematic views illustrating an example of amethod for operating the information processor.

FIG. 10A, FIG. 10B1, and FIG. 10B2 are schematic views illustrating anexample of a method for operating the information processor.

FIG. 11 is a diagram showing AnoGAN, which can be applied to oneembodiment of the present invention.

MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described below. Note thatone embodiment of the present invention is not limited to the followingdescription, and it will be readily appreciated by those skilled in theart that modes and details of the present invention can be modified invarious ways without departing from the spirit and scope of the presentinvention. One embodiment of the present invention therefore should notbe construed as being limited to the following description of theembodiments.

Note that in the drawings attached to this specification, the blockdiagram in which components are classified according to their functionsand shown as independent blocks is illustrated; however, it is difficultto separate actual components completely according to their functions,and one component may be related to a plurality of functions or aplurality of components may achieve one function.

Embodiment 1

In this embodiment, an information processing system of one embodimentof the present invention and an information processing method using theinformation processing system will be described. With the informationprocessing system of one embodiment of the present invention and theinformation processing method, fatigue, drowsiness, or the like of auser of an information terminal such as a smartphone or a tablet can bepresumed. In particular, eye fatigue of the user of the informationterminal can be detected.

<Structure Example of Information Processing System>

FIG. 1 is a block diagram illustrating a structure example of aninformation processing system 10 that is the information processingsystem of one embodiment of the present invention. The informationprocessing system 10 includes an information processor 20 and aninformation processor 30.

The information processor 20 includes an imaging unit 21, a display unit22, an arithmetic unit 23, a main memory unit 24, an auxiliary memoryunit 25, and a communication unit 26. Data or the like can betransmitted between components included in the information processor 20via a transmission path 27. The information processor 30 includes animaging unit 31, a display unit 32, an arithmetic unit 33, a main memoryunit 34, an auxiliary memory unit 35, and a communication unit 36. Dataor the like can be transmitted between components included in theinformation processor 30 via a transmission path 37.

The imaging unit 21 and the imaging unit 31 have a function ofperforming image capturing and obtaining imaging data. The display unit22 and the display unit 32 have a function of displaying an image.

The arithmetic unit 23 and the arithmetic unit 33 have a function ofperforming arithmetic processing. The arithmetic unit 23 has a functionof performing predetermined arithmetic processing, for example, on datatransmitted from the imaging unit 21, the main memory unit 24, theauxiliary memory unit 25, or the communication unit 26 to the arithmeticunit 23 via the transmission path 27. The arithmetic unit 33 has afunction of performing predetermined arithmetic processing, for example,on data transmitted from the imaging unit 31, the main memory unit 34,the auxiliary memory unit 35, or the communication unit 36 to thearithmetic unit 33 via the transmission path 37. The arithmetic unit 23and the arithmetic unit 33 have a function of performing arithmeticoperation by machine learning. The arithmetic unit 23 and the arithmeticunit 33 have a function of performing arithmetic operation using aneural network, for example. The arithmetic unit 23 and the arithmeticunit 33 can include a CPU (Central Processing Unit) and a GPU (GraphicsProcessing Unit), for example.

The main memory unit 24 and the main memory unit 34 have a function ofstoring data, a program, and the like. The arithmetic unit 23 canexecute arithmetic processing by reading the data, the program, and thelike stored in the main memory unit 24. The arithmetic unit 23, forexample, can execute predetermined arithmetic processing on the dataread from the main memory unit 24 by executing the program read from themain memory unit 24. The arithmetic unit 33 can execute arithmeticprocessing by reading the data, the program, and the like stored in themain memory unit 34. The arithmetic unit 33, for example, can executepredetermined arithmetic processing on the data read from the mainmemory unit 34 by executing the program read from the main memory unit34.

The main memory unit 24 and the main memory unit 34 preferably operateat higher speed than the auxiliary memory unit 25 and the auxiliarymemory unit 35. For example, the main memory unit 24 and the main memoryunit 34 can include a DRAM (Dynamic Random Access Memory), an SRAM(Static Random Access Memory), or the like.

The auxiliary memory unit 25 and the auxiliary memory unit 35 have afunction of storing data, a program, and the like for a longer periodthan the main memory unit 24 and the main memory unit 34. The auxiliarymemory unit 25 and the auxiliary memory unit 35 can include an HDD (HardDisk Drive), an SSD (Solid State Drive), or the like, for example.Furthermore, the auxiliary memory unit 25 and the auxiliary memory unit35 may include a nonvolatile memory such as an ReRAM (Resistive RandomAccess Memory, also referred to as a resistance-change memory), a PRAM(Phase change Random Access Memory), an FeRAM (Ferroelectric RandomAccess Memory), an MRAM (Magnetoresistive Random Access Memory, alsoreferred to as a magneto-resistive memory), or a flash memory.

The communication unit 26 has a function of transmitting and receivingdata or the like to/from a device or the like provided outside theinformation processor 20. The communication unit 36 has a function oftransmitting and receiving data or the like to/from a device or the likeprovided outside the information processor 30. For example, it ispossible to supply data or the like from the information processor 20 tothe information processor 30 by supplying data or the like from thecommunication unit 26 to the communication unit 36. Furthermore, thecommunication unit 26 and the communication unit 36 can have a functionof supplying data or the like to a network and a function of obtainingdata or the like from the network.

Here, in the case where the arithmetic unit 23 and the arithmetic unit33 have a function of performing arithmetic operation by machinelearning, for example, the arithmetic unit 23 can perform learning andthe learning result can be supplied from the information processor 20 tothe information processor 30. For example, in the case where thearithmetic unit 23 and the arithmetic unit 33 have a function ofperforming arithmetic operation using a neural network, the arithmeticunit 23 can obtain a weighting coefficient or the like by performinglearning, and the weighting coefficient or the like can be supplied fromthe information processor 20 to the information processor 30. In theabove manner, even when the arithmetic unit 33 provided in theinformation processor 30 does not perform learning, inference on datathat has been input to the arithmetic unit 33 can be performed on thebasis of a learning result by the arithmetic unit 23 provided in theinformation processor 20. Accordingly, the arithmetic throughput of thearithmetic unit 33 can be lower than that of the arithmetic unit 23.

In the case where the arithmetic unit 23 performs learning and thelearning result is supplied from the information processor 20 to theinformation processor 30, the information processor 20 can be providedin a server, for example. Note that in the case where the informationprocessor 20 is provided in the server, the imaging unit 21 and thedisplay unit 22 are not necessarily provided in the informationprocessor 20. That is, the imaging unit 21 and the display unit 22 maybe provided outside the information processor 20.

The information processor 30 can be provided in an information terminalsuch as a smartphone, a tablet, or a personal computer, for example. Atleast a part of the components of the information processor 20 and atleast a part of the components of the information processor 30 may beboth provided in the server. For example, the arithmetic unit 23 and thearithmetic unit 33 may be provided in the server. In this case, forexample, data obtained by the information terminal is supplied to thearithmetic unit 33 via a network and the arithmetic unit 33 provided inthe server performs inference or the like on the data. Then, theinference result is supplied to the information terminal via thenetwork, whereby the information terminal can obtain the inferenceresult.

<Example of Information Processing Method>

An example of an information processing method using the informationprocessing system 10 will be described below. Specifically, an exampleof a method for presuming fatigue, drowsiness, or the like of the userof the information terminal provided with the information processor 30included in the information processing system 10 by arithmetic operationusing machine learning will be described.

FIG. 2 and FIG. 3 are flow charts showing an example of a method forpresuming fatigue, drowsiness, or the like by arithmetic operation usingmachine learning. Learning is shown in FIG. 2 and inference is shown inFIG. 3.

An example of a learning method will be described with reference to FIG.2 and the like. First, the imaging unit 21 captures a moving image. Forexample, the imaging unit 21 captures a moving image including a humanface (Step S01). Here, the moving image refers to a group of images oftwo or more frames. Although details will be described later, learningdata is produced on the basis of the moving image captured by theimaging unit 21 and the arithmetic unit 23 performs learning. Thus, forexample, in the case where the imaging unit 21 captures a moving imageincluding a human face, it is preferable that the imaging unit 21capture a moving image for a large number of people that differ in sex,race, physical constitution, and the like.

Note that image processing may be performed on the moving image capturedby the imaging unit 21. For example, noise removal, gray-scaletransformation, normalization, contrast adjustment, and the like can beperformed. Furthermore, binarization or the like may be performed on theimages included in the moving image. By such processing, a later stepcan be performed with high accuracy. For example, detection of a firstobject performed in Step S02, which will be described later, can beperformed with high accuracy.

Next, the arithmetic unit 23 detects a first object from each of thecaptured images. For example, in the case where a moving image of ahuman face is captured in Step S01, the first object can be an eye (StepS02). The first object, for example, can be detected with a cascadeclassifier.

The detection can be performed with, for example, Haar Cascades. Notethat in the case where the first object is an eye and both eyes areincluded in one image, only one of the eyes can be detected.

After that, the arithmetic unit 23 detects a second object from each ofthe detected first objects. For example, when the first object is aneye, the second object can be a pupil (Step S03). The pupil can bedetected from the eye by circular extraction, for example. Details ofthe method for detecting the pupil from the eye will be described later.

Here, the pupil is a hole surrounded by an iris and can be referred toas a “black part of the eye.” The pupil has a function of adjusting theamount of light entering a retina. The iris is a thin film positionedbetween a cornea and a lens and can be regarded as a colored portion inthe eye, for example.

Next, the arithmetic unit 23 calculates each size of the detected secondobjects (Step S04). For example, in the case where the second object isdetected by circular extraction, the radius or diameter of the secondobject can be regarded as the size of the second object. In the casewhere the shape of the second object is extracted as an ellipticalshape, the length of the major axis and the length of the minor axis canbe regarded as the size of the second object. The area of the secondobject can be regarded as the size of the second object.

Then, the arithmetic unit 23 performs learning using the size of thesecond object to obtain the learning result (Step S05). Specifically,the learning result is obtained on the basis of a change over time inthe size of the second object. The learning can be performed using aneural network, for example. In this case, the learning result can be aweighting coefficient or the like as described above. Details of thelearning method will be described later.

Next, the information processor 20 supplies the learning result to theinformation processor 30 (Step S06). Specifically, the learning resultobtained by the arithmetic unit 23 is transmitted to the communicationunit 26 via the transmission path 27 and then supplied from thecommunication unit 26 to the communication unit 36. The learning resultsupplied to the communication unit 36 can be stored in the auxiliarymemory unit 35. In addition, the learning result may be stored in theauxiliary memory unit 25.

Next, an example of an inference method on the basis of the learningresult obtained by the method shown in the FIG. 2 or the like will bedescribed with reference to FIG. 3 and the like. First, the imaging unit31 captures a moving image. For example, the imaging unit 31 captures amoving image including the face of the user of the information terminalprovided with the information processor 30 (Step S11). Note that in thecase where image processing is performed on the moving image captured bythe imaging unit 21 in Step S01 shown in FIG. 2, similar imageprocessing is preferably performed on the moving image captured by theimaging unit 31, in which case inference can be performed with highaccuracy.

Next, the arithmetic unit 33 detects a first object from each of imagesincluded in the captured moving image. For example, in the case where amoving image of a human face is captured in Step S11, the first objectcan be an eye (Step S12). The first object can be detected by a methodsimilar to the detection method used in Step S02 shown in FIG. 2.

After that, the arithmetic unit 33 detects a second object from each ofthe detected first objects. For example, when the first object is aneye, the second object can be a pupil (Step S13). The second object canbe detected by a method similar to the detection method used in Step S03shown in FIG. 2.

Next, the arithmetic unit 33 calculates each size of the detected secondobjects (Step S14). A method similar to that used in Step SO4 shown inFIG. 2 can be used as the method for calculating the size.

Then, the arithmetic unit 33 to which the learning result obtained bythe arithmetic unit 23 in Step S05 shown in FIG. 2 has been inputperforms inference on the basis of a change over time in the size of thesecond object. For example, in the case where the face of the user ofthe information terminal provided with the information processor 30 isincluded in the moving image captured by the imaging unit 31 and thesecond object is the pupil of the eye of the user, the arithmetic unit33 can presume fatigue, drowsiness, or the like of the user (Step S15).Details of the inference method will be described later.

Note that the size of the pupil changes depending on not only whetherthere is fatigue, drowsiness, or the like, but also, for example, thebrightness of the environment. Therefore, for example, a plurality ofmoving images are preferably captured for the face of the same personunder various brightness of the environment in Step S01 shown in FIG. 2.Thus, fatigue, drowsiness, or the like of the user of the informationterminal provided with the information processor 30 can be presumed withhigh accuracy regardless of, for example, the brightness of theenvironment.

In one embodiment of the present invention, the information processor 30having a function of presuming fatigue, drowsiness, or the like asdescribed above is provided in an information terminal such as asmartphone, a tablet, or a personal computer. This makes it possible todetect fatigue, drowsiness, or the like of the user of the informationterminal in real time without using a dedicated device.

[Example of Method for Detecting Pupil]

Next, an example of the method for detecting a pupil performed in StepS03 and Step S13 will be described. FIG. 4 is a flow chart showing anexample of the method for detecting a pupil.

First, the arithmetic unit obtains an image 41 that is an imageincluding the detected eye (Step S31). FIG. 5A is a schematic viewillustrating Step S31. As illustrated in FIG. 5A, the arithmetic unitobtains the image 41 including the eye detected from the image capturedby the imaging unit. Specifically, in Step S03, the arithmetic unit 23obtains, as the image 41, the image including the eye detected in StepS02 by the arithmetic unit 23. Furthermore, in Step S13, the arithmeticunit 33 obtains, as the image 41, the image including the eye detectedin Step S12 by the arithmetic unit 33. Note that when the image capturedby the imaging unit is a color image, the arithmetic unit may convertthe image 41 into a gray-scale image after the arithmetic unit obtainthe image 41.

Next, the arithmetic unit obtains an image 42 by performing expansionprocessing on the image 41, and then performs contraction processing toobtain an image 43 (Step S32). That is, closing processing is performedon the image 41, whereby the image 43 is obtained. FIG. 5B is aschematic view illustrating the expansion processing and the contractionprocessing.

After that, the arithmetic unit subtracts the image 43 from the image 41to obtain an image 44 (Step S33). That is, the image 44 is an imageexpressed by a difference between the image 41 and the image 43. In StepS33, the arithmetic unit can obtain the image 44 by performing Black-hatconversion using the image 41 and the image 43.

Next, the arithmetic unit adds together the image 41 obtained in StepS31 and the image 44 obtained in Step S33 to obtain an image 45 (StepS34). Note that in the case where the image 41 is converted into agray-scale image in Step S31, the image 41 after gray-scaletransformation and the image 44 can be added together in Step S34.

Note that all or part of the processing shown in Step S32 to Step S34 isnot necessarily performed. Alternatively, processing other than theprocessing shown in Step S32 to Step S34 may be performed.

Then, the arithmetic unit performs image processing on the image 45 toobtain an image 46 (Step S35). For example, the arithmetic unit performsprocessing such as noise removal and smoothing on the image 45.Furthermore, processing such as edge detection and binarization isperformed. Specifically, for example, noise removal by a median valuefilter and smoothing by a Gaussian filter are performed on the image 45,and then edge detection by a Canny method and binarization processingare performed. Note that the noise removal may be performed by a movingaverage filter, for example. The smoothing may be performed by a movingaverage filter or a median filter, for example. Furthermore, the edgedetection may be performed by a Laplacian filter.

Next, the arithmetic unit detects an iris 47 from the image 46. The iris47 can be detected using Hough transform, for example. In the case ofusing Hough transform, the iris 47 can be detected as a circular shape,for example. Alternatively, the iris 47 can be detected as an ellipticalshape, for example. Note that the iris 47 may be detected usinggeneralized Hough transform.

Then, the arithmetic unit obtains an image 49 including the detectediris 47 (Step S36). For example, the image 49 is extracted from theimage 46 on the basis of the coordinates of the detected iris 47 in theimage 46.

FIG. 5C is a schematic view illustrating Step S36. For example, theimage 49 can be a rectangle whose four sides are in contact with theiris 47 as illustrated in FIG. 5C. For example, in the case where theiris 47 is detected as a circular shape, the image 49 can be a squarewhose four sides are in contact with the iris 47. Note that each side ofthe image 49 is not necessarily in contact with the iris 47. Forexample, an image with a predetermined number of pixels, in which theiris 47 is positioned in the center, may be the image 49.

Next, the arithmetic unit detects a pupil 48 from the image 49 (StepS37). The pupil 48 is detected from the image 49 by arithmetic operationusing a neural network, for example.

Step S37 is performed using a generator on which learning has beenperformed in advance. Here, the generator is a program performingarithmetic operation by machine learning and has a function ofoutputting data corresponding to input data. Specifically, learningenables the generator to make an inference on the data input to thegenerator.

FIG. 6A is a schematic view illustrating the above-described learning.Here, a generator that performs learning is referred to as a generator50. In the case where a neural network is used as the generator 50, aconvolutional neural network (CNN) can be used as the generator 50. Inparticular, U-net as one kind of CNN is preferably used. In U-net, aninput image is subjected to down-sampling by convolution, and thensubjected to up-sampling by deconvolution using a feature value obtainedby the down-sampling. The arithmetic unit 23 and the arithmetic unit 33can have a function of the generator 50.

Learning of the generator 50 can be performed by supervised learningusing data 51 and data 52. The data 51 can be a group of images 59. Theimage 59 includes an iris 57 and a pupil 58. The image 59 can beobtained by the information processor 20 by a method similar to Step S01and Step S02 shown in FIG. 2 and Step S31 to Step S36 shown in FIG. 4.Note that the imaging unit 21 captures the moving image of the face inStep S01; in the case where the image 59 is obtained, however, theimaging unit 21 does not necessarily capture the moving image. Forexample, the imaging unit 21 may capture one (one-frame) image for eachperson.

The data 52 is data indicating the coordinates of the pupil 58 includedin the image 59. Specifically, the data 52 can be a binary image inwhich the pupil 58 portion has a different color from the other portion.The data 52 can be obtained, for example, by filling the pupil 58included in the image 59. Alternatively, the data 52 can be obtained insuch a manner that an image including an eye is obtained by a methodsimilar to Step S31 shown in FIG. 4, and then the pupil 58 included inthe image including the eye is filled.

The learning of the generator 50 is performed so that output data canbecome closer to the data 52 when the data 51 is input to the generator50. That is, the learning of the generator 50 is performed using thedata 52 as correct data. By the learning of the generator 50, thegenerator 50 generates a learning result 53. In the case where a neuralnetwork is used as the generator 50, the learning result 53 can be aweighting coefficient or the like.

The learning of the generator 50, that is, the generation of thelearning result 53 can be performed by, for example, the arithmetic unit23 included in the information processor 20. Then, when the learningresult 53 is supplied from the information processor 20 to theinformation processor 30, the arithmetic unit 33 can also performinference similar to that of the arithmetic unit 23. The learning result53 generated by the arithmetic unit 23 can be stored in the auxiliarymemory unit 25, for example. In addition, the learning result 53generated by the arithmetic unit 23 and supplied to the informationprocessor 30 can be stored in the auxiliary memory unit 35, for example.

Through the above steps, the learning of the generator 50 is terminated.

FIG. 6B is a schematic view illustrating Step S37. That is, FIG. 6B is aschematic view illustrating detection of the pupil 48 from the image 49.

As illustrated in FIG. 6B, in Step S37, the image 49 obtained by thearithmetic unit in Step S36 is input to the generator 50 in which thelearning result 53 has been read. Thus, the generator 50 can performinference on the image 49 and output data indicating the coordinates ofthe pupil 48. For example, the generator 50 can output a binary image inwhich the color of the pupil 48 is made different from the color of theother portion.

By the above-described method, in Step S03 or Step S13, the pupil can bedetected from the eye detected in Step S02 or Step S12.

Detection of a pupil by arithmetic operation using machine learning canbe performed in a shorter time than that in the case where, for example,a pupil is visually detected. Furthermore, for example, even when thepupil reflects a surrounding landscape, the pupil can be detected withhigh accuracy.

Note that the method for detecting the pupil 48 in Step S37 is notlimited to the method illustrated in FIG. 6A and FIG. 6B. For example,the image 49 may be a color image, and after the color image 49 issubjected to gray-scale transformation, edge detection of the pupil 48may be performed. Then, after the edge detection, the pupil 48 may bedetected.

The gray-scale transformation of the image 49 can be performed usingpartial least squares (PLS) regression, for example. By the gray-scaletransformation of the image 49, a difference between the brightness ofthe pupil 48 and the brightness of the iris 47 can be large.Accordingly, the boundary between the pupil 48 and the iris 47 can beemphasized; thus, the edge detection of the pupil 48 can be performedwith high accuracy. Therefore, the pupil 48 can be detected with highaccuracy.

The edge detection of the pupil 48 can be performed by, for example, aCanny method or a Laplacian filter. Furthermore, detection of the pupil48 after the edge detection can be performed using Hough transform, forexample. In the case of using Hough transform, the pupil 48 can bedetected as a circular shape, for example. Alternatively, the pupil 48can be detected as an elliptical shape, for example. Note that the pupil48 may be detected using generalized Hough transform.

In the case where not only the iris but also the pupil is detected inStep S03 or Step S13, image capturing can be performed using infraredrays in Step S01 or Step S11. An iris reflects infrared rays. Incontrast, a pupil does not reflect infrared rays. Therefore, the irisand the pupil can be clearly distinguished from each other by imagecapturing using infrared rays in Step S01 or Step S11. Thus, the pupilcan be detected with high accuracy.

[Example_1 of Method for Presuming Fatigue, Drowsiness, or the Like]

Next, an example of a method for presuming fatigue, drowsiness, or thelike of the user of the information terminal provided with theinformation processor 30 by arithmetic operation using machine learningwill be described. Specifically, an example of the learning method usingthe size of a pupil, which is performed in Step SO5, will be described.In addition, an example of the method for presuming fatigue, drowsiness,or the like by inference based on the learning result, which isperformed in Step S15, will be described. Note that the followingdescription is made on the assumption that the second object is a pupil.

FIG. 7A1 is a schematic view illustrating Step S05. In Step S05,learning of a generator 60 that is a program performing arithmeticoperation by machine learning is performed. A neural network can be usedas the generator 60. Time series data such as a change over time in thesize of a pupil is input to the generator 60, the details of which willbe described later. Thus, in the case where a neural network is used asthe generator 60, a recurrent neural network (RNN) is preferably used asthe generator 60. Alternatively, a long short-term memory (LSTM) ispreferably used as the generator 60. Alternatively, a gated recurrentunit (GRU) is preferably used.

The learning of the generator 60 can be performed using data 61 and data62. The data 61 is the data obtained in Step SO4 and can be a changeover time in the size of the pupil. As described above, for example, inthe case where the pupil is detected by circular extraction, the radiusor diameter of the pupil can be regarded as the size of the pupil. Inaddition, in the case where the pupil is extracted as an ellipticalshape, the length of the major axis and the length of the minor axis canbe regarded as the size of the pupil. Moreover, the area of the pupilcan be regarded as the size of the pupil. In FIG. 7A1, a change overtime in the size of the pupil from Time 1 to n-1 (n is an integergreater than or equal to 3) is regarded as the data 61.

The data 61 may be a change over time in the ratio between the size ofthe pupil and the size of the iris. In this case, it is preferable thatthe iris and the pupil be extracted as the same kind of shapes. Forexample, when the iris is extracted as a circular shape, the pupil isalso extracted as a circular shape. When the iris is extracted as anelliptical shape, the pupil is also extracted as an elliptical shape.When the data 61 is the change over time in the ratio between the sizeof the pupil and the size of the iris is detected, for example, by themethod shown in Step S37, the resolutions of the images 49 including theiris 47 and the pupil 48 can differ from each other. For example, theresolution of the image 49 including the iris 47 and the pupil 48 of afirst person and the resolution of the image 49 including the iris 47and the pupil 48 of a second person can differ from each other.

The data 62 is the size of the pupil at Time n. That is, the data 62 isthe size of the pupil at a time after the time when the size of thepupil included in the data 61 is measured. Note that in the case wherethe data 61 is the change over time in the ratio between the size of thepupil and the size of the iris, the data 62 is also the ratio betweenthe size of the pupil and the size of the iris.

FIG. 7A2 is a diagram showing an example of the relation between thepupil diameter and the time. In FIG. 7A2, a black circle represents anactual measured value of the pupil diameter. Also in the other diagrams,an actual measured value is represented by a black circle in some cases.As shown in FIG. 7A2, the data 62 can be the size of the pupil at thetime after the time when the size of the pupil included in the data 61is measured. For example, the data 62 can be the size of the pupil at atime subsequent to the time when the size of the pupil included in thedata 61 is measured last.

Here, in the case where a function of presuming whether there is fatigueis imparted to the generator 60, a change over time in the size of apupil of a person who has fatigue is not included in the data 61 or thedata 62. That is, the data 61 is a change over time in the size of apupil of a person who has no fatigue, and the data 62 is the size of thepupil of the person who has no fatigue. In the case where a function ofpresuming whether there is drowsiness is imparted to the generator 60, achange over time in the size of a pupil of a person who has drowsinessis included in neither the data 61 nor the data 62. That is, the data 61is a change over time in the size of a pupil of a person who does nothave drowsiness, and the data 62 is the size of a pupil of a person whohas no drowsiness.

The learning of the generator 60 is performed so that output data canbecome closer to the data 62 when the data 61 is input to the generator60. That is, the learning of the generator 60 is performed using thedata 62 as correct data. By the learning of the generator 60, thegenerator 60 generates a learning result 63. In the case where a neuralnetwork is used as the generator 60, the learning result 63 can be aweighting coefficient or the like.

FIG. 7B1 and FIG. 7B2 are schematic views illustrating Step S15 and showan example of the method for presuming fatigue, drowsiness, or the likeof the user of the information terminal provided with the informationprocessor 30, with the use of the generator 60. In Step S15, first, asillustrated in FIG. 7B1, data 64 indicating the change over time in thesize of the pupil, which is obtained in Step S14, is input to thegenerator 60 in which the learning result 63 has been read. For example,in the case where the change over time in the size of the pupil fromTime 1 to n-1 are used as the input data at the time of the learning ofthe generator 60 in Step S05, the change over time in the size of thepupil from Time 1 to n-1 are also used as input data at the time of aninference in Step S15. That is, the data 64 is a change over time in thesize of the pupil of the user of the information terminal provided withthe information processor 30 from Time 1 to n-1. Thus, the generator 60performs inference on the data 64 to output data 65. Note that,furthermore, data from Time 2 to n is used as input data and data atTime n+1 may be inferred with the use of the data 65 that is theinference data at Time n.

Note that in the case where the data 61 is the change over time in theratio between the size of the pupil and the size of the iris, the data64 is also a change over time in the ratio between the size of the pupiland the size of the iris. When the data 64 is the change over time inthe ratio between the size of the pupil and the size of the iris, theresolutions of the images 49 including the iris 47 and the pupil 48 candiffer from each other, for example, in the case of detecting the pupilby a method shown in Step S37. For example, the resolution of the image49 which the arithmetic unit obtains in order to calculate the ratiobetween the size of the pupil 48 and the size of the iris 47 at Time 1can be different from the resolution of the image 49 which thearithmetic unit obtains in order to calculate the ratio between the sizeof the pupil 48 and the size of the iris 47 at Time n-1.

The data 65 is an estimated value of the size of the pupil at a timeafter the time when the size of the pupil included in the data 64 ismeasured, which is obtained by the calculation by performing inferenceon the data 64 on the basis of the learning result 63. For example, inthe case where the data 64 is a change over time in the size of thepupil from Time 1 to n-1, the data 65 can be the size of the pupil atTime n. In FIG. 7B1, the actual measured values of the size of the pupilfrom Time 1 to n-1 are represented as xi to xn-i, respectively. Inaddition, an estimated value of the size of the pupil at Time n isrepresented as xn(E). Note that in the case where the data 64 is thechange over time in the ratio between the size of the pupil and the sizeof the iris, the data 65 is the ratio between the size of the pupil andthe size of the iris.

Next, as illustrated in FIG. 7B2, data 66 that represents the actualmeasured value of the size of the pupil at, for example, Time n and thedata 65 that is the data output from the generator 60 are compared witheach other. In other words, for example, the actual measured value atTime n and the estimated value are compared with each other. With thecomparison result, whether there is fatigue, drowsiness, or the like ispresumed. For example, in the case where the generator 60 has a functionof presuming whether there is fatigue, the learning of the generator 60is performed with the use of a change over time in the size of a pupilof a person who has no fatigue, as input data, for example. Accordingly,when the user of the information terminal provided with the informationprocessor 30 is in a state without fatigue, the data 65 becomes close tothe data 66. That is, a difference between the data 65 and the data 66becomes small. In contrast, when the user of the information terminalprovided with the information processor 30 is in a state with fatigue,the difference between the data 66 and the data 65 becomes larger thanthat in the case where the user of the information terminal providedwith the information processor 30 is in the state without fatigue. Fromthe above, the comparison between the data 66 and the data 65 makes itpossible to presume whether the user of the information terminalprovided with the information processor 30 has fatigue. The same appliesto the case of presuming whether there is drowsiness. Note that in thecase where the data 65 is an estimated value of the ratio between thesize of the pupil and the size of the iris, the data 66 is an actualmeasured value of the ratio between the size of the pupil and the sizeof the iris.

The function of the generator 60 can be imparted to both the arithmeticunit 23 and the arithmetic unit 33. In this case, the arithmetic unit 23included in the information processor 20 can generate the learningresult 63 by performing the learning of the generator 60, and thelearning result 63 can be supplied from the information processor 20 tothe information processor 30. Thus, even without learning, thearithmetic unit 33 provided in the information processor 30 can performinference on the data input to the arithmetic unit 33, on the basis ofthe learning result by the arithmetic unit 23 provided in theinformation processor 20. Accordingly, the arithmetic processingperformance of the arithmetic unit 33 can be lower than that of thearithmetic unit 23. Note that the learning result 63 can be stored inthe auxiliary memory unit 25 and the auxiliary memory unit 35.

[Example_2 of Method for Presuming Fatigue, Drowsiness, or the Like]

FIG. 8A and FIG. 8B are schematic views illustrating Step S05 and showan example of a learning method of a generator, which is different fromthe above method. Specifically, FIG. 8A illustrates an example of amethod for generating data input to a generator as learning data, andFIG. 8B illustrates an example of a learning method of a generator 80.The generator 80 is a program that performs arithmetic operation bymachine learning. For example, a neural network can be used as thegenerator 80.

Data 81 shown in FIG. 8A is data obtained in Step SO4 and can be achange over time in the size of a pupil. As described above, forexample, in the case where the pupil is detected by circular extraction,the radius or diameter of the pupil can be regarded as the size of thepupil. In addition, in the case where the pupil is extracted as anelliptical shape, the length of the major axis and the length of theminor axis can be regarded as the size of the pupil. Moreover, the areaof the pupil can be regarded as the size of the pupil. As in thelearning method shown in FIG. 7A1, the data 81 can be a change over timein the ratio between the size of the pupil and the size of an iris.

Here, the data 81 is subjected to Fourier transform to generate data 82.As shown in FIG. 8A, the change over time in the pupil diameter can beconverted into frequency characteristics of the pupil diameter by theFourier transform. Note that in the case where the data 81 is a changeover time in the ratio between the size of the pupil and the size of theiris, the data 82 can be frequency characteristics of the ratio betweenthe size of the pupil and the size of the iris.

The learning of the generator 80 can be performed using the data 82 anddata 83 as illustrated in FIG. 8B. The data 82 represents the frequencycharacteristics of the pupil diameter as described above. The data 83can be a label indicating whether there is fatigue. For example,frequency characteristics of the size of a pupil of a person who hasfatigue and frequency characteristics of the size of a pupil of a personwho has no fatigue are made to be both included as the data 83. Then,the frequency characteristics of the size of the pupil of the person whohas fatigue is linked with a label “with fatigue” and the frequencycharacteristics of the size of the pupil of the person who has nofatigue is linked with a label “without fatigue.” The data 83 may be alabel indicating whether there is drowsiness.

The learning of the generator 80 is performed so that output data canbecome closer to the data 83 when the data 82 is input to the generator80. That is, the learning of the generator 80 is performed using thedata 83 as correct data. By the learning of the generator 80, thegenerator 80 generates a learning result 84. In the case a neuralnetwork is used as the generator 80, the learning result 84 can be aweighting coefficient or the like.

The change over time in the size of the pupil is subjected to Fouriertransform, whereby the data input to the generator 80 can become datathat is not time series data. Therefore, the generator 80 can performlearning and inference without using an RNN as the generator 80.

FIG. 9A and FIG. 9B are schematic views illustrating Step S15 and showan example of the method for presuming fatigue, drowsiness, or the likeof the user of the information terminal provided with the informationprocessor 30, with the use of the generator 80.

Data 85 shown in FIG. 9A is data obtained in Step SO4 and can be achange over time in the size of the pupil. As described above, when thepupil is detected by, for example, circular extraction, the radius ordiameter of the pupil can be regarded as the size of the pupil. Inaddition, in the case where the pupil is extracted as an ellipticalshape, the length of the major axis and the length of the minor axis canbe regarded as the size of the pupil. Moreover, the area of the pupilcan be regarded as the size of the pupil. Note that in the case wherethe data 81 shown in FIG. 8A is a change over time in the ratio betweenthe size of the pupil and the size of the iris, the data 85 is also achange over time in the ratio between the size of the pupil and the sizeof the iris.

Here, the data 85 is subjected to Fourier transform to generate data 86.As shown in FIG. 9A, the change over time in the pupil diameter can beconverted into frequency characteristics of the pupil diameter byFourier transform. Note that in the case where the data 85 is a changeover time in the ratio between the size of the pupil and the size of theiris, the data 86 can be frequency characteristics of the ratio betweenthe size of the pupil and the size of the iris.

Then, as shown in FIG. 9B, the data 86 after Fourier transform is inputto the generator 80. Thus, the generator 80 can perform inference on thedata 86 and output data 87 indicating whether there is fatigue. Notethat in the case where the data 87 shown in FIG. 9B is a labelindicating whether there is drowsiness, the data 87 output by thegenerator 80 can be data indicating whether there is drowsiness.

As in the case of the function of the generator 60 and the function of agenerator 70, the function of the generator 80 can be imparted to boththe arithmetic unit 23 and the arithmetic unit 33. Thus, the arithmeticprocessing performance of the arithmetic unit 33 can be lower than thatof the arithmetic unit 23.

[Example_3 of Method for Presuming Fatigue, Drowsiness, or the Like]

FIG. 10A is a schematic view illustrating Step S05 and shows an exampleof a learning method of a generator, which is different from the abovemethod. In FIG. 10A, learning of the generator 70 is performed. Thegenerator 70 is a program that performs arithmetic operation by machinelearning. For example, a neural network can be used as the generator 70;for example, an autoencoder can be used.

When the generator 70 performs learning, data 71 is input to thegenerator 70. The data 71 is data obtained in Step SO4 and can be achange over time in the size of a pupil. Here, in the case where afunction of presuming whether there is fatigue is imparted to thegenerator 70, a change over time in the size of a pupil of a person whohas fatigue is not included in the data 71. That is, the data 71 is achange over time in the size of a pupil of a person who has no fatigue.In the case where a function of presuming whether there is drowsiness isimparted to the generator 70, a change over time in the size of a pupilof a person who has drowsiness is not included in the data 71. That is,the data 71 is a change over time in the size of a pupil of a person whohas no drowsiness.

As described above, for example, in the case where the pupil is detectedby circular extraction, the radius or diameter of the pupil can beregarded as the size of the pupil. In addition, in the case where thepupil is extracted as an elliptical shape, the length of the major axisand the length of the minor axis can be regarded as the size of thepupil. Moreover, the area of the pupil can be regarded as the size ofthe pupil.

As in the learning method shown in FIG. 7A1, the data 71 can be a changeover time in the ratio between the size of the pupil and the size of theiris. As in the case shown in FIG. 8A, a change over time in the size ofthe pupil that has been subjected to Fourier transform may be used asthe data 71.

The learning of the generator 70 is performed so that data 72 that isoutput data can become closer to the input data 71 when the data 71 isinput to the generator 70. That is, the learning of the generator 70 isperformed so that the data 71 and the data 72 can be equal to eachother. By the learning of the generator 70, the generator 70 generates alearning result 73. In the case a neural network is used as thegenerator 70, the learning result 73 can be a weighting coefficient orthe like.

FIG. 10B1 and FIG. 10B2 are schematic views illustrating Step S15 andshow an example of the method for presuming fatigue, drowsiness, or thelike of the user of the information terminal provided with theinformation processor 30, with the use of the generator 70. In Step S15,first, as illustrated in FIG. 10B1, data 74 indicating a change overtime in the size of the pupil obtained in Step S14 is input to thegenerator 70 to which the learning result 73 is read. Thus, thegenerator 70 performs inference on the data 74 to output data 75.

Note that in the case where the data 71 is the change over time in theratio between the size of the pupil and the size of the iris, the data74 is also a change over time in the ratio between the size of the pupiland the size of the iris. In the case where data after Fourier transformis used as the data 71, data after Fourier transform is also used as thedata 74. For example, in the case where the data 71 is a change overtime in the size of the pupil that has been subjected to Fouriertransform, the data 74 is also a change over time in the pupil that hasbeen subjected to Fourier transform.

Next, as illustrated in FIG. 10B2, the data 74 that is the data input tothe generator 70 and the data 75 that is the data output from thegenerator 70 are compared with each other. With the comparison result,whether there is fatigue, drowsiness, or the like is presumed. Forexample, in the case where the generator 70 has a function of presumingwhether there is fatigue, the learning of the generator 70 is performedwith the use of a change over time in the size of a pupil of a personwho has no fatigue, as input data, for example. Accordingly, when theuser of the information terminal provided with the information processor30 is in a state without fatigue, the data 75 that is the output datafrom the generator 70 becomes close to the data 74 that is the inputdata to the generator 70. That is, a difference between the data 74 andthe data 75 becomes small. In contrast, when the user of the informationterminal provided with the information processor 30 is in a state withfatigue, the difference between the data 74 and the data 75 becomeslarger than that in the case where the user of the information terminalprovided with the information processor 30 is in the state withoutfatigue. From the above, the comparison between the data 74 and the data75 makes it possible to presume whether the user of the informationterminal provided with the information processor 30 has fatigue. Thesame applies to the case of presuming whether there is drowsiness.

As in the case of the function of the generator 60, the function of thegenerator 70 can be imparted to both the arithmetic unit 23 and thearithmetic unit 33. Thus, the arithmetic processing performance of thearithmetic unit 33 can be lower than that of the arithmetic unit 23.

[Example_4 of method for presuming fatigue, drowsiness, or the like]

The learning performed in Step S05 and the inference performed in StepS15 on the basis of the learning result may be performed using agenerative adversarial network (GAN). They may be performed using AnoGAN(Anormaly GAN), for example. FIG. 11 illustrates AnoGAN that enables theabove learning and inference.

The AnoGAN illustrated in FIG. 11 includes a generator 91 and adiscriminator 92. The generator 91 and the discriminator 92 can beformed with a network.

Data 93 that is time-series data representing a change over time in thesize of a pupil of a person who has no fatigue, drowsiness, or the likeand is obtained by image capturing is input to the discriminator 92.Alternatively, data 95 that is time-series data generated by thegenerator 91 to which data 94 is input is input to the discriminator 92.The discriminator 92 has a function of making a determination (alsoreferred to as an authenticity determination) whether the input data isthe data 93 obtained by image capturing or the data 95 generated by thegenerator 91. Note that the data 93 may be data obtained in such amanner that the time-series data that represents a change over time inthe size of a pupil of a person who has no fatigue, drowsiness, or thelike and is obtained by image capturing is subjected to Fouriertransform.

The determination result is output as data 96. The data 96 can becontinuous values between 0 and 1, for example. In this case, forexample, the discriminator 92 outputs a value close to 1 as the data 96in the case where data input after the termination of the learning isthe data 93 obtained by image capturing and outputs a value close to 0as the data 96 in the case where the input data is the data 95 generatedby the generator 91.

The data 94 is multi-dimensional random numbers (also referred to as alatent variable). Here, the latent variable represented by the data 94is referred to as a latent variable z. The generator 91 has a functionof generating data that is as close as possible to the data representinga change over time in the size of the pupil of the person who has nofatigue, drowsiness, or the like, on the basis of such data 94.

As for the learning, the learning of the discriminator 92 and thelearning of the generator 91 are performed alternately. That is, theweighting coefficient of the neural network forming the generator 91 isfixed at the time of the learning of the discriminator 92. In addition,the weighting coefficient of the neural network forming thediscriminator 92 is fixed at the time of the learning of the generator91.

The data 93 obtained by image capturing or the data 95 generated by thegenerator 91 is input to the discriminator 92 at the time of thelearning of the discriminator 92. A correct label is given to the datainput to the discriminator 92. The correct label for the data 96 outputby the discriminator 92 can be determined in the following manner. Forexample, the correct label is “1” when the data 93 is input to thediscriminator 92, and the correct label is “0” when the data 95 is inputto the discriminator 92. The learning performed by the above methodenables the discriminator 92 to make an authenticity determination.

The data 94 representing the latent variable z is input to the generator91 at the time of the learning of the generator 91. Then, the generator91 generates the data 95 on the basis of the input data 94. A correctlabel of the data 96 is “1”. Then, the learning of the generator 91 isperformed so that the value of the data 96 output from the discriminator92 can be “1”. As the learning of the generator 91 advances, thegenerator 91 becomes able to generate data similar to the data 93obtained by image capturing, as the data 95.

After the completion of the learning of the generator 91, the generator91 becomes able to generate the data 95 similar to the data 93 obtainedby image capturing even when any latent variable z is input as the data94.

Next, operation at the time of inference is described.

First, data representing a change over time in the size of a pupil of aperson who has no fatigue, drowsiness, or the like is obtained bypicture taking. At this time, a space of a latent variable is searched,and a latent variable zl generating data that is the most similar to thedata of the size of the pupil of the person who has no fatigue,drowsiness, or the like is found by a gradient descent method or thelike. The generator is 91 and has a function of generating dataextremely similar to the data representing a change over time in thesize of the pupil of the person who has no fatigue, drowsiness, or thelike by learning. Thus, data generated from the latent variable zl bythe generator and the data representing a change over time in the sizeof the pupil of the person who has no fatigue, drowsiness, or the likeare extremely similar to each other.

Next, data representing a change over time in the size of a pupil of aperson who has fatigue, drowsiness, or the like is obtained by imagecapturing. At this time, the space of the latent variable is searched,and a latent variable z2 generating data that is the closest to the datarepresenting a change over time in the size of the pupil of the personwho has fatigue, drowsiness, or the like is found by a gradient descentmethod or the like. The generator 91 has a capability of generating dataextremely close to the data representing a change over time in the sizeof the pupil of the person who has no fatigue, drowsiness, or the likebut does not have a capability of generating data similar to the datarepresenting a change over time in the size of the pupil of the personwho has fatigue, drowsiness, or the like. Therefore, the data generatedfrom the latent variable z2 by the generator and the data obtained bypicture taking and representing a change over time in the size of thepupil of the person who has fatigue, drowsiness, or the like are notextremely similar to each other. From the above, whether there isfatigue, drowsiness, or the like can be presumed by the generator 91.

As described above, as shown in FIG. 7 to FIG. 11, the informationprocessor 30 presumes fatigue, drowsiness, or the like of the user ofthe information terminal provided with the information processor 30 byarithmetic operation using machine learning. With the use of machinelearning, it is possible to presume fatigue, drowsiness, or the likewith high accuracy even when, for example, a change over time in afeature value in the case of presuming that there is fatigue and achange over time in a feature value in the case of presuming that thereis no fatigue are not set manually. Specifically, it is possible topresume fatigue, drowsiness, or the like with high accuracy even when,for example, a change over time in the size of the pupil in the case ofpresuming that there is fatigue and a change over time in the size ofthe pupil in the case of presuming that there is no fatigue are not setmanually. In addition, fatigue, drowsiness, or the like can be presumedby a simple method because it is possible to presume fatigue,drowsiness, or the like even when, for example, a change over time in afeature value in the case of presuming that there is fatigue and achange over time in a feature value in the case of presuming that thereis no fatigue are not set manually.

An alarm indicating that fatigue, drowsiness or the like is generatedcan be displayed, for example, on a display portion of the informationterminal provided with the information processor 30 when the informationprocessor 30 presumes that there is fatigue, drowsiness, or the like.Accordingly, it becomes possible to urge the user of the informationterminal to stop the use of the information terminal early, for example.Alternatively, the information terminal provided with the informationprocessor 30 can be turned off. Therefore, it is possible to inhibithealth hazards that are caused when the user of the information terminalcontinues using the information terminal even though fatigue,drowsiness, or the like is caused in the user.

REFERENCE NUMERALS

10: information processing system, 20: information processor, 21:imaging unit, 22: display unit, 23: arithmetic unit, 24: main memoryunit, 25: auxiliary memory unit, 26: communication unit, 27:transmission path, 30: information processor, 31: imaging unit, 32:display unit, 33: arithmetic unit, 34: main memory unit, 35: auxiliarymemory unit, 36: communication unit, 37: transmission path, 41: image,42: image, 43: image, 44: image, 45: image, 46: image, 47: iris, 48:pupil, 49: image, 50: generator, 51: data, 52: data, 53: learningresult, 57: iris, 58: pupil, 59: image, 60: generator, 61: data, 62:data, 63: learning result, 64: data, 65: data, 66: data, 70: generator,71: data, 72: data, 73: learning result, 74: data, 75: data, 80:generator, 81: data, 82: data, 83: data, 84: learning result, 85: data,86: data, 87: data, 91: generator, 92: discriminator, 93: data, 94:data, 95: data, 96: data

1. An information processor comprising: an imaging unit and anarithmetic unit having a function performing arithmetic operation bymachine learning, wherein the imaging unit has a function of obtaining amoving image that is a group of images of two or more frames, whereinthe arithmetic unit has a function of detecting a first object from eachof two or more of the images included in the moving image, wherein thearithmetic unit has a function of detecting a second object from each ofthe detected first objects, wherein the arithmetic unit has a functionof calculating a size of each of the detected second objects, andwherein the arithmetic unit has a function of performing learning usinga change over time in the size of the second object.
 2. The informationprocessor according to claim 1, wherein the machine learning isperformed with a neural network.
 3. The information processor accordingto claim 1, wherein the moving image comprises a face, the first objectis an eye, and the second object is a pupil.
 4. An information processorhaving a function of performing inference on the basis of a learningresult obtained by performing learning using a change over time in asize of a first object shown in two or more first images included in afirst moving image, wherein the information processor has a function ofobtaining a second moving image, wherein the information processor has afunction of detecting a second object from each of two or more secondimages included in the second moving image, wherein the informationprocessor has a function of detecting a third object from each of thedetected second objects, wherein the information processor has afunction of calculating a size of each of the detected third objects,and wherein the information processor has a function of performinginference on a change over time in the size of the third object on thebasis of the learning result.
 5. The information processor according toclaim 4, wherein the learning and the inference are performed with aneural network, and wherein the learning result comprises a weightingcoefficient.
 6. The information processor according to claim 4, whereinthe first moving image comprises a first face, the second moving imagecomprises a second face, the first and third objects are pupils, and thesecond object is an eye.
 7. The information processor according to claim6, wherein a function of presuming fatigue of a person comprising thesecond face is included.